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[Document Name] SPECIFICATION 
[Title of the Invention] 

Image Processing Apparatus, Image Processing Method and 
Recording Medium 
[Claims] 

[Claim 1] An image processing apparatus comprising: 

data acquiring means for acquiring image data representing an image 
composed of a plurality of pixels each having one color component of one of N 
fundamental colors; 

class determining means for determining a class of the first image data, 
from the color component of the most dense pixel of the pixels of the image data; 
and 

operation means for calculating the color component of each of the N 
fundamental colors, for one of the pixels represented by the image data, in 
accordance with the class determined by the class determining means. 

[Claim 2] The image processing apparatus according to claim 1, further 
comprising: 

a prediction tap determining means for determining a prediction tap from 
the pixels represented by the image data, said prediction tap predicting a prescribed 
pixel; and 

memory means for storing prediction coefficients for predicting the 
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prescribed pixel of the image signal, 

wherein the operation means performs calculation on the prediction tap 
determined by the prediction tap determining means and the prediction coefficient 
corresponding to the class determined by the class determining means. 

[Claim 3] The image processing apparatus according to claim 2, wherein the 
tap determining means determines, as the prediction tap, pixels of all N 
fundamental colors. 
[Claim 4] An image processing method comprising : 

a data acquiring step of acquiring image data representing an image 
composed of a plurality of pixels each having one color component of one of N 
fundamental colors; 

a class determining step of determining a class of the image data, from the 
color component of the most dense pixel of the pixels of the image data; and 

an operation step of calculating the color component of each of the N 
fundamental colors, for one of the pixels represented by the image data, in 
accordance with the class determined in the class determining step. 

[Claim 5] A recording medium for providing a computer-readable program 
for use in an image processing apparatus, said program describing: 

a data acquiring step of acquiring image data representing an image 
composed of a plurality of pixels each having one color component of one of N 
fundamental colors; 
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a class determining step of determining a class of the image data, from the 
color component of the most dense pixel of the pixels of the image data; and 

an operation step of calculating the color component of each of the N 
fundamental colors, for one of the pixels represented by the image data, in 
accordance with the class determined in the class determining step. 
[Claim 6] An image processing apparatus comprising: 

data acquiring means for acquiring first image data representing an image 
composed of a plurality of pixels each having one color component of one of N 
fundamental colors; 

data generating means for generating, form the first image data, second 
image data of lower resolution than the first image data; 

class determining means for determining a class of the second image data, 
from the color component of the most dense pixel of the pixels of the second image 
data; and 

operation means for calculating a coefficient for obtaining the color 
component of each of the N fundamental colors, for one of the pixels represented 
by third image data, from the first image data acquired by the data acquiring means, 
the second image data generated by the data generating means and the third image 
data representing an image composed of a plurality of pixels each having one color 
component of one of N fundamental colors and based on the class determined by 
the class determining means. 
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[Claim 7] An image processing method comprising: 

a data acquiring step of acquiring first image data representing an image 
composed of a plurality of pixels each having one color component of one of N 
fundamental colors; 

a data generating step of generating, form the first image data, second 
image data of lower resolution than the first image data; 

a class determining step of determining a class of the second image data, 
from the color component of the most dense pixel of the pixels of the second image 
data; and 

an operation step of calculating a coefficient for obtaining the color 
component of each of the N fundamental colors, for one of the pixels represented 
by third image data, from the first image data acquired in the data acquiring step, 
the second image data generated by the data generating means and the third image 
data representing an image composed of a plurality of pixels each having one color 
component of one of N fundamental colors and based on the class determined by 
the class determining step. 

[Claim 8] A recording medium for providing a computer-readable program 
for use in an image processing apparatus, said program describing: 

a data acquiring step of acquiring first image data representing an image 
composed of a plurality of pixels each having one color component of one of N 
fundamental colors; 
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a data generating step of generating, form the first image data, second 
image data of lower resolution than the first image data; 

a class determining step of determining a class of the second image data, 
from the color component of the most dense pixel of the pixels of the second image 
data; and 

an operation step of calculating a coefficient for obtaining the color 
component of each of the N fundamental colors, for one of the pixels represented 
by third image data, from the first image data acquired in the data acquiring step, 
the second image data generated by the data generating means and the third image 
data representing an image composed of a plurality of pixels each having one color 
component of one of N fundamental colors and based on the class determined by 
the class determining step. 
[Detailed Description of the Invention] 
[0001] 

[Technical Field of the Invention] 

The present invention relates to an image processing apparatus, an image 
processing method and a recording medium. More particularly, the invention 
relates to a an image processing apparatus, an image processing method and a 
recording medium, which can provide images of higher resolution. 

[0002] 

[Prior Art] 
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A single-plate CCD is often used as an imaging device in digital still 
cameras or the like, in order to render the cameras smaller. The single-plate CCD 
outputs data representing n x m pixels, each having one of the three primary colors, 
R, G and B, as is illustrated in FIG. 1 . Which color one pixel has depends on the 
color filter array that is placed in front of the CCD. 
[0003] 

A pixel arranged at the back of a G filter, for example, outputs an 
G-component image signal, but cannot output a R-component image signal or a B- 
component image signal. Likewise, a pixel outputs positioned at the back of an R 
filter outputs only an R-component image signal, but cannot output a G-component 
image signal or a B-component image signal. A pixel arranged at the back of a B 
filter outputs only a B-component image signal, but cannot output an R-component 
image signal or a G-component image signal. 
[0004] 

To perform various processes on the image data, however, R, G and B 
color components are necessary for each pixel. Classification-adaptation process, 
for example, is therefore performed to obtain image data equivalent to the output 
of a three-plate CCD, from the image data items output by the single-plate CCD. 
Thus, n x m R-pixel data items, n x m G-pixel data items and n x m B-pixel data 
items are obtained. 
[0005] 
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[Object of the Invention] 

Hitherto, the R pixels, G pixels and B pixels are independently selected as 
class taps, thereby to obtain image data equivalent to the output of a three-plate 
CCD, from the image data items output by the single-plate CCD. That is, to 
predict an R pixel, the R pixel is used as a class tap; to predict a G pixel, the G 
pixel is used as a class tap; and to predict a G pixel, the B pixel is used as a class 
tap. 

[0006] 

Consequently, only one R pixel exists for every four pixels, an only one B 
pixel for every four pixels, among the n x m pixels, if the color filter array 
incorporated in the signal-plate CCD has the Bayer arrangement as shown in FIG. 
1. It is therefore impossible to effect the classification-adaptation process with 
high efficiency. 

The present invention has been made in view of the foregoing. The object 
of the invention is to perform the classification-adaptation process with high 
efficiency. 
[0007] 

[Means to Solve the Problem] 

The image processing apparatus as defined in claim 1 comprises: data 
acquiring means for acquiring image data representing an image composed of a 
plurality of pixels each having one color component of one of N fundamental 
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colors; class determining means for determining a class of the first image data, 
from the color component of the most dense pixel of the pixels of the image data; 
and operation means for calculating the color component of each of the N 
fundamental colors, for one of the pixels represented by the image data, in 
accordance with the class determined by the class determining means. 
[0008] 

The image processing method defined in claim 4 comprises: a data 
acquiring step of acquiring image data representing an image composed of a 
plurality of pixels each having one color component of one of N fundamental 
colors; a class determining step of determining a class of the image data, from the 
color component of the most dense pixel of the pixels of the image data; and an 
operation step of calculating the color component of each of the N fundamental 
colors, for one of the pixels represented by the image data, in accordance with the 
class determined in the class determining step. 
[0009] 

The recording medium described in claim 5 provides a computer-readable 
program for use in an image processing apparatus. The program describing: a 
data acquiring step of acquiring image data representing an image composed of a 
plurality of pixels each having one color component of one of N fundamental 
colors; a class determining step of determining a class of the image data, from the 
color component of the most dense pixel of the pixels of the image data; and an 
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operation step of calculating the color component of each of the N fundamental 
colors, for one of the pixels represented by the image data, in accordance with the 
class determined in the class determining step. 
[0010] 

The image processing apparatus defined in claim 6 comprises: data 
acquiring means for acquiring first image data representing an image composed of 
a plurality of pixels each having one color component of one of N fundamental 
colors; data generating means for generating, form the first image data, second 
image data of lower resolution than the first image data; class determining means 
for determining a class of the second image data, from the color component of the 
most dense pixel of the pixels of the second image data; and operation means for 
calculating a coefficient for obtaining the color component of each of the N 
fundamental colors, for one of the pixels represented by third image data, from the 
first image data acquired by the data acquiring means, the second image data 
generated by the data generating means and the third image data representing an 
image composed of a plurality of pixels each having one color component of one 
of N fundamental colors and based on the class determined by the class 
determining means. 
[0011] 

The image processing method described in claim 7 comprises: a data 
acquiring step of acquiring first image data representing an image composed of a 
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plurality of pixels each having one color component of one of N fundamental 
colors; a data generating step of generating, form the first image data, second 
image data of lower resolution than the first image data; a class determining step of 
determining a class of the second image data, from the color component of the 
most dense pixel of the pixels of the second image data; and an operation step of 
calculating a coefficient for obtaining the color component of each of the N 
fundamental colors, for one of the pixels represented by third image data, from the 
first image data acquired in the data acquiring step, the second image data 
generated by the data generating means and the third image data representing an 
image composed of a plurality of pixels each having one color component of one 
of N fundamental colors and based on the class determined by the class 
determining step. 
[0012] 

The recording medium defined in claim 8 provides a computer-readable 
program for use in an image processing apparatus. The program describing: a 
data acquiring step of acquiring first image data representing an image composed 
of a plurality of pixels each having one color component of one of N fundamental 
colors; a data generating step of generating, form the first image data, second 
image data of lower resolution than the first image data; a class determining step of 
determining a class of the second image data, from the color component of the 
most dense pixel of the pixels of the second image data; and an operation step of 
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calculating a coefficient for obtaining the color component of each of the N 
fundamental colors, for one of the pixels represented by third image data, from the 
first image data acquired in the data acquiring step, the second image data 
generated by the data generating means and the third image data representing an 
image composed of a plurality of pixels each having one color component of one 
of N fundamental colors and based on the class determined by the class 
determining step. 
[0013] 

In the image processing apparatus, image processing method and recording 
medium, which are described in claims 1, 4 and 5, respectively, the class is 
determined from the color component of the most dense pixel. 
[0014] 

In the image processing apparatus, image processing method and recording 
medium, which are described in claims 6, 7 and 8, respectively, a prediction 
coefficient is calculated from the class determined from the color component of the 
most dense pixel. 
[0015] 

[Preferred Embodiment of the Invention] 

FIG. 2 illustrates the principles of processing an output of a single-plate 
CCD, generating image data having a density higher than the CCD output (in this 
embodiment, four times as high as the CCD output). As shown in FIG. 2, image 
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data of 2n x 2m R pixels, image data of 2n x 2m R pixels and image data of 2n x 
2m B pixels are generated directly from the image data of n x m pixels, which has 
been output by the single-plate CCD. 
[0016] 

FIG. 2 shows a digital still camera designed to picking up an image, by 
virtue of the 1 of the operating principles described above. In the camera shown 
in FIG. 2, the lens 1 focuses the light reflected from the picked up image. The 
light is applied to the single-plate CCD 3 through an iris 2. The CCD 3 converts 
the input light, and the output of the CCD 3 is supplied to an AGC (Automatic 
Gain Control/CDS (Correlated Double Sampling) circuit 4. 
[0017] 

The AGC/CDS circuit 4 sets the input signal at a prescribed level, outputs 
the signal at the prescribed level and removes the 1/f noise generated in the CCD 3. 
The AGC/CDS circuit 4 performs the function of an electronic shutter, under the 
control of a main CPU (Central Processing Unit) 8. The signal output from the 
AGC/CDS circuit 4 is input to an A/D converter 5 and converted to a digital signal, 
which is input to an image-signal processing circuit 6. The image-signal 
processing circuit 6 performs, on the input signal, a defect-eliminating process, a 
digital clamping process, a white-balance adjusting process, a gamma-correction 
process, an interpolation process using the classification-adaptation process. A 
memory 12 is connected to the image-signal processing circuit 6. The memory 
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12 stores image data, whenever necessary, while the image-signal processing 
circuit 6 is performing these processes. 
[0018] 

A timing generator (TG) 7 is controlled by the main CPU 8. It generates 
various timing signals. The timing signals are output to the CCD 3, AGC/CDS 
circuit 4, AID converter 45, main CPU 8 and the like. 
[0019] 

A motor 9 drives the iris 2 under the control of the main CPU 8. The 
amount of light applied from the lens 1 to the CCD 3 is controlled. A motor 10 
which is controlled by the main CPU 8 controls the lens 1 , to control the state 
of the focus for the CCD3 of the lens 1 . Under the controlled by the main CPU 8, 
a light-emitting section 1 1 applies a prescribed amount of flash light to the object 
of photography. 
[0020] 

A storage-medium interface (I/F) 13 is provided. The interface 13 
supplies the image data output from the image-signal processing circuit 6, 
whenever necessary, thus storing the image data into the memory 14. After 
effecting a prescribed interface process, the interface 13 supplies the image data to 
a storage medium 15, storing the image data therein. The user can easily inserted 
the storage medium 15 into the digital still camera and remove the same therefrom. 
[0021] 
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A controller 16 is provided to control the image-signal processing circuit 6 
and the storage-medium interface 13 under the control of the main CPU 8. 
Various user commands are input from a terminal 20 to the main CPU 8. A 
power-supply section 17 is provided, which incorporates a battery 18 and a DC/DC 
converter 19. The DC/DC converter 19 converts the power supplied from the 
battery 18 to a DC voltage of a predetermined value. The DC voltage is applied 
to the components of the camera. The battery 18 is a rechargeable one. The 
user can insert the battery 1 8 into and remove it from the digital still camera. 
[0022] 

The process of picking up the image and storing the data representing the 
image of the object into the storage medium 15 will be described, with reference to 
the flowchart of FIG. 3. 

In Step SI, the lens 1 collects the light reflected from the object. The 
light is applied through the iris 2 to the CCD 3. At this time, the main CPU 8 
controls the motor 9, driving the iris 2. The amount of light applied to the CCD 3 
is thereby adjusted to the predetermined value. The main CPU 8 controls a motor 
10, too. The motor 10 adjusts the position of the lens 1, accomplishing focus 
control. 
[0023] 

The main CPU 8 determines whether the terminal 20 has received a signal 
that is generated when the user operates the release button (not shown) provided on 
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the camera. If the user has not operated the release button, the operation returns 
to Step SI, which is repeated. If it is determined in Step S2 that the release button 
has been operated, the main CPU 8 drives the light-emitting section 11, which 
emits flash light. The flash light illuminates the picked up image. At this time, 
the main CPU 8 controls the timing generator 7, which generates various timing 
signals. In Step S3, the CCD 3 converts the light reflected from the image to an 
electric signal in synchronism with the timing signal supplied from the timing 
generator 7. The electric signal is output to the AGC/CDS circuit 4. 
[0024] 

In Step S4, the AGC/CDS circuit 4 removes the 1/f noise component of the 
signal input from the CCD 3. The signal, now having no noise component, is 
output as a level having a prescribed level, to the A/D converter 5. In Step S5, 
the A/D converter 5 converts the signal output from the AGC/CDS circuit 4, to a 
digital image signal. The digital image signal is output to the image-signal 
processing circuit 6. In Step S6, the image-signal processing circuit 6 stores the 
image signal into the memory 12, if necessary, and processes the image signal. 
The image signal, thus processed, is output to the storage-medium interface 13. 
The storage-medium interface 1 3 stores the image signal input from the 
image-signal processing circuit 6 and stores the same, if necessary, into the 
memory 14. After performing the interface process, the interface 13 supplies the 
image signal to the storage medium 15. In Step S7, the storage medium 15 stores 
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the image signal supplied to it. 
[0025] 

FIG. 4 shows an example of the image-signal processing circuit 6. The 
image signal, which has been generated by the single-plate CCD 3 and input from 
the A/D converter 5, is input to a defect-correcting circuit 31. The 
defect-correcting circuit 3 1 performs the process of eliminating a defective part of 
the image signal, which corresponds to a pixel or pixels of the CCD 3 that fail to 
respond to the input light or to a pixel or pixels that always hold an electric charge. 
[0026] 

The A/D converter 5 shifts the signal value a little in the positive direction, 
in order to prevent the negative value from being cut. A clamping circuit 32 
shifts the signal output from the defect-correcting circuit 31, back in the opposite 
direction. A white-balancing (WB) circuit 33 corrects the gains of the R, G and B 
signals input from the clamping circuit 32. A gamma-correcting circuit 34 
corrects the value of the signal input from the white-balancing circuit 33, in 
accordance with a gamma curve. 
[0027] 

An interpolating section 35 is provided, which performs a 
classification-adaptation process, thereby interpolating the signal output from the 
gamma-correcting circuit 34. The interpolating section 35 has an ADRC 
(Adaptive Dynamic Range Control) process circuit 41 that processes the signal 
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input from the gamma-correcting circuit 34. The ADRC process circuit 41 
performs an ADRC process on the signal value of the class tap contained in the 
input signal. A classification circuit 42 determines a class from the signal value 
of the class tap ADRC-processed by the ADRC process circuit 41 , thereby 
outputting a class number. An adaptation process circuit 43 reads from a 
coefficient memory 44 the coefficient that corresponds to the class input from the 
classification circuit 42. The circuit 43 multiplies the coefficient by the pixel 
value of a prediction tap for the image signal input through the ADRC process 
circuit 41 and classification circuit 42, thereby calculating a necessary pixel value. 
[0028] 

A correction circuit 36 is provided, which performs correction, such as 
edge correction, on the signal input from the adaptation process circuit 43 of the 
interpolating section 35, thus improving the visual feature of the image. An RGB 
matrix circuit 37 outputs the RGB signal input from the correction circuit 36 
without processing these signals. Alternatively, the circuit 37 multiplies this 
signal by a prescribed conversion matrix, thereby converting the signals to a YUV 
signal or the like. The YUV signal or the like is output from the circuit 37. 
[0029] 

How the image-signal processing circuit 6 operates will be explained, with 
reference to the flowchart of FIG. 5. In Step SI 1, the defect-correcting circuit 31 
corrects a defective pixel or pixels, if any, contained in the image signal input from 
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the A/D converter 5. For example, the circuit 32 replaces any pixel that does not 
respond to light or any pixel that always has an electric charge, with the average 
value of the neighboring pixels. In Step SI 2, the clamping circuit 32 performs a 
clamping process on the signal input from the defect-correcting circuit 3 1 , thereby 
to correct the offset value set in the A/D converter 5. In Step SI 3, the 
white-balancing circuit 33 adjusts the levels of the R signal, G signal and B signal 
to such levels that these signals may represent white. In Step SI 4, the 
gamma-correcting circuit 34 carries out the gamma-correction process. 
[0030] 

In Step SI 5, the ADRC process circuit 41 divides the input signal to p x q 
blocks. In Step SI 6, the circuit 41 extracts a class tap for each block and 
performs the ADRC process on the class tap. 
[0031] 

In Step SI 7, the classification circuit 42 performs the classification process 
on the data input from the ADRC process circuit 41. That is, the circuit 42 
determines the class that corresponds to the value obtained by ADRC-processing 
the class tap. The class thus determined is output to the adaptation process circuit 
43. In Step SI 8, the adaptation process circuit 43 reads the prediction coefficient 
corresponding to the class input from the classification circuit 42, from the 
coefficient memory 44, and multiplies the prediction coefficient by the 
corresponding prediction tap, thereby generating the data representing a predicted 
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pixel. 
[0032] 

Next, in Step SI 9, the interpolating section 35 determines whether the 
interpolation has been effected on all blocks. If any block has not been subjected 
to the interpolation, the operation returns to Step SI 6. In this case, Steps S16 et 
seq. are repeated. 
[0033] 

If it is determined in Step S19 that all blocks have been subjected to the 
interpolation, the operation advances to Step S20. In Step S20 the correction 
circuit 36 carries out correction, such as edge correction, on the input signal. In 
Step S21, the RGB matrix circuit 37 performs an operation, if necessary, to convert 
the RGB data to YUB data (that is, to convert the input data to a color space), and 
outputs the result of conversion. 

Specific examples of the class tap and prediction tap will be described. 
As shown in FIG. 6, for example, a B pixel may be the pixel to be predicted. 
Then, as shown in FIG. 7, the class tap is composed of eight G pixels. Of these 
eight G pixels, four are arranged above, below, to the left of and to the right of, the 
pixel to be predicted, respectively. Two are arranged at upper-left and lower-left 
positions with respect to the G pixel on the left of the pixel to be predicted, and the 
remaining two are arranged at upper-right and lower-right positions with respect to 
the G pixel on the right of the pixel to be predicted. In this case, the prediction 
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tap is composed of 5 x 5 pixels, including R pixels, G pixels and B pixels, as 
illustrated in FIG. 8, with a B pixel located at the center of the 5 x 5 matrix. 
[0034] 

An R pixel may be the pixel to be predicted as is illustrated in FIG. 9. In 
this case, the class tap is composed of eight G pixels as shown in FIG. 10. Of 
these eight G pixels, four are arranged above, below, to the left of and to the right 
of, the R pixel to be predicted, respectively. Two are arranged at upper-left and 
lower-left positions with respect to the G pixel on the left of the pixel to be 
predicted, and the remaining two are arranged at upper-right and lower-right 
positions with respect to the G pixel on the right of the pixel to be predicted. The 
prediction tap in this case is composed of 25 pixels, including R pixels, G pixels 
and B pixels, as illustrated in FIG. 1 1 , with an R pixel located at the center of the 5 
x 5 matrix. 
[0035] 

Further, a G pixel may be the pixel to be predicted as is illustrated in FIG. 
12. In this case, the class tap is composed of nine G pixels as shown in FIG. 13. 
More precisely, the class tap shown in FIG. 13 is composed of the G pixel to be 
predicted, four G pixels arranged above, below, to the left of and to the right of, the 
G pixel to be predicted, respectively, two G pixels arranged above and below the G 
pixel to be predicted and spaced therefrom by a R pixel, and two G pixels arranged 
to left and right of the G pixel to be predicted and spaced therefrom by a B pixel. 
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The prediction tap in this case is composed of 5 x 5 pixels that have R, G and B 
color components. Of these pixels, one G pixel is to be predicted, and the 
remaining pixels surround the G pixel, as is illustrated in FIG. 14. 
[0036] 

A luminance signal Y is generated from the signal values for R, G 
and B, as in indicated by the following equation: 
[0037] 

Y = 0.30R + 0.59G + 0.11B 

As seen from this equation, the G component more influences the 
luminance signal Y than the other color components. Therefore, as shown in FIG. 
6, G pixels are arranged more densely than the R and B pixels in the Bayer 
arrangement. 
[0038] 

In view of this, it may be possible to effect the classification-adaptation 
process with higher precision if the class tap is composed of only G pixels of the 
image signal. 
[0039] 

FIG. 15 shows a learning apparatus that acquires prediction coefficients to 
be stored in the coefficient memory 44 and to be used in the adaptation process 
described above. In the learning apparatus 61, digital image data is supplied, as a 
teacher image, to an extraction circuit 71 and a teacher-image block generating 
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circuit 75. The teacher image has, for example, a density four times as high as the 
original image data. The extraction circuit 71 extracts pixels from the 
teacher-image data, on the assumption that color filters, each having a 
magnification opposite to the one the photographing system should have. As a 
result, the extraction circuit 71 outputs student image data that is equivalent to the 
image data the single-place CCD 3 has generated. 
[0040] 

The image data representing the student image is input to a student-image 
block generating circuit 72. The student-image block generating circuit 72 
extracts the class tap to be used in the classification process and prediction tap to 
be used in prediction process, from the student-image data generated by the 
extraction section 71, while referring to the relation between the teacher-image 
signal and the predicted pixel for each block. An ADRC process circuit 73 
performs an ADRC process on the prediction tap of the pixels input from the 
student-image block generating circuit 72, and outputs the pixels to a classification 
circuit 74. The classification circuit 74 determines the class from the value of the 
class tap that has been ADRC-processed by the ADRC process circuit 73. The 
class is output to an operating circuit 76. The operating circuit 76 receives the 
image data of the prediction tap from the student- image block generating circuit 72 
through the ADRC process circuit 73 and classification circuit 74. 
[0041] 
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The teacher-image block generating circuit 75 extracts the predicted pixel 
from the teacher-image data, while referring to the relation between the class tap in 
the student image. The predicted pixel extracted is output to the operating circuit 
76. The operating section 76 carries out an operation on the class number 
supplied from the classification circuit 74, while maintaining the relation between 
the prediction tap supplied from the classification circuit 74 and the predicted 
image supplied from the teacher-image block generating circuit 75. Thus, the 
operating circuit 76 generates the data of a normal equation. The data of the 
normal equation, generated by the operating section 76, is stored into a learned 
data memory 77. The memory 77 sequentially stores the matrix coefficients of 
the normal equation, supplied from the operating circuit 76. Another operating 
circuit 78 is provided, which performs least square method, thus solving the 
normal equation by using the matrix of the normal equation stored in the learned 
data memory 77. Prediction coefficients are thereby calculated for one class. 
The prediction coefficients are stored into a coefficient memory 79. 
[0042] 

How the learning apparatus 61 functions will be explained with reference 
to the flowchart of FIG. 16. In Step S31, the teacher- image block generating 
circuit 75 converts the input teacher-image signal to a block, thus extracting a 
predicted pixel. The predicted pixel is output to the operating circuit 76. In Step 
S32, the extraction section 71 extracts the teacher-image data, thereby generating 
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student-image data corresponding to the image data output by the single-plate CCD 
3. The teacher-image data is output to the student-image block generating circuit 
72. In Step S33, the student-image block generating circuit 72 converts the input 
student-image data to a block. The circuit 72 generates a class tap and a 
prediction tap for the block. 
[0043] 

In Step S34, the ADRC process circuit 73 carries out the ADRC process on 
the prediction tap supplied from the student-image block generating circuit 72. In 
Step S3 5, the classification circuit 74 determines a class code on the basis of the 
class tap supplied from the ADRC process circuit 73. 
[0044] 

In Step S3 6, the operating circuit 76 generates the normal equation for the 
class number supplied from the classification circuit 74 3 on the basis of the 
prediction tap supplied from the classification circuit 74 and the predicted image 
supplied from the teacher-image block generating circuit 75. In Step S37, it is 
determined whether the operating circuit 76 has finished processing all blocks or 
not. If there are any blocks not processed yet, the operation returns to Step S3 6. 
In this case, Step S36 et seq. are repeated. If it is determined in Step S37 that all 
blocks have been processed, the operation goes to Step S3 8. In Step S3 8, the 
operating circuit 78 solves the normal equation stored in the learned data memory 
77 by means of, for example, the least square method. In Step S3 9, it is 
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determined whether or not the operating circuit 78 has solved the normal equation 
for all classes. If the equation has not been solved for any classes, the operation 
returns to Step S3 8. In this case, Step S3 8 et seq. are repeated. 
[0045] 

If it is determined in Step S3 9 that the normal equation has been solved for 
all classes, the operation is terminated. The prediction coefficients thus obtained 
are supplied to the coefficient memory 79 and stored therein. The prediction 
coefficients are then stored into the coefficient memory 44 shown in FIG. 4. 
[0046] 

In the embodiment, the color filters of the CCD 3 are of the Bayer 
arrangement illustrated in FIG. 17(A). Instead, the color filters may be of 
interline arrangement shown in FIG. 17(B), a G-stripe, RB chessboard arrangement 
shown in FIG. 17(C), a G-stripe, RB perfect chessboard arrangement depicted in 
FIG. 17(D), a stripe arrangement illustrated in FIG. 17(E), a slant stripe 
arrangement shown in FIG. 17(F), or a primary color-difference arrangement 
illustrated in FIG. 17(G). 
[0047] 

The filters provided for the CC3 may be complement-color filters, not 
primary-color filters. In this case, each pixel has a value selected from yellow 
(Ye), cyan (Cy), magenta (Mg) and green (G), and the color filter may be of a field 
interleave arrangement or a strip arrangement, both shown in FIG. 18. 
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[0048] 

In order to evaluate the operating efficiency of the embodiment described 
above, the inventors conducted simulation, using a color-filter array of the Bayer 
arrangement. 
[0049] 

An image signal equivalent to an output of a three-plate CCD was 
subjected to an extraction operation in which the magnification of the 
classification-adaptation process was applied and the positional relation of pixels 
was taken into account. An image signal equivalent to an output of the 
single-plate CCD was thereby generated. The output of the single-plate CCD was 
converted to an image signal having twice as many pixels in both the low direction 
and the column direction, by means of a classification-adaptation process in which 
the prediction coefficients for the CCD output were utilized. 
[0050] 

Further, the nine high-vision images of the IT standard were used as 
teacher images. The class taps and prediction taps used were those illustrated in 
FIG. 7, FIG. 8, FIG. 10 and FIGS. 13 and 14. 
[0051] 

The simulation resulted in an image that was more sharp at edges and had a 
higher resolution than in the case where the class tap applied to predict an R pixel 
(alternatively, a G signal or a B signal) is a pixel that has R, G and B components 
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mixed together. Simulation was conducted by means of linear interpolation, not 
classification-adaptation process. The image generated by classification process 
was found superior to the image signal provided by linear interpolation in terms of 
resolution and S/N ratio. 
[0052] 

In the embodiment described above, the output of the single-plate CCD is 
converted to an image that is equivalent to one output by a three-plate CCD. 
Nevertheless, the present invention can be applied to generate image data of any 
other density. 
[0053] 

Moreover, the present invention can be applied not only to a digital still 
camera, but also to a video camera and any other type of an image-processing 
apparatus. 
[0054] 

The recording medium storing the computer program is presented to users, 
in the form of a magnetic disk, a CD-ROM, solid memory or the like. Instead, 
the program can be transmitted to users through networks and 
digital-communications satellites. 
[0055] 

[Advantageous Effect of the Invention] 

As has been described, the class is determined from the color component of 
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the most dense pixel in the image processing apparatus, image processing method 
and recording medium, which are described in claims 1, 4 and 5, respectively. 
The N fundamental color components are calculated from the class thus 
determined. Therefore, image data can be generated with high precision. 
[0056] 

In the image processing apparatus, image processing method and recording 
medium, which are described in claims 6, 7 and 8, respectively, coefficients for 
obtaining N fundamental color components are calculated from the third image 
data representing an image that is composed of pixels having the N fundamental 
color components, in accordance with the class determined from the color 
component of the most dense pixel. Hence, it is possible to provide coefficients 
that help to generate image data with high precision. 
[Brief Description of the Drawings] 
[FIG. 1] 

A diagram illustrating the principles of generating an output equivalent to 
one generated by a three-plate CCD. 
[FIG. 2] 

A block diagram of a digital still camera according to the present invention. 
[FIG. 3] 

A flowchart explaining the operation of the digital still camera shown in 

FIG. 2. 
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[FIG. 4] 

A block diagram showing the image-signal processing circuit 6 of the 
camera shown in FIG. 2. 
[FIG. 5] 

A flowchart explaining how the image-signal processing circuit of FIG. 4 
operates. 
[FIG. 6] 

A diagram for explaining a predicted pixel. 
[FIG. 7] 

A diagram illustrating a class tap. 
[FIG. 8] 

A diagram explaining a prediction tap. 
[FIG. 9] 

A diagram showing a predicted pixel 
[FIG. 10] 

A diagram depicting a class tap. 
[FIG. 11] 

A diagram explaining a prediction tap. 
[FIG. 12] 

A diagram showing a predicted pixel. 
[FIG. 13] 
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A diagram depicting a class tap. 
[FIG. 14] 

A diagram showing a prediction tap. 
[FIG. 15] 

A block diagram of a learning apparatus that generates prediction 
coefficients to be stored into the coefficient memory 44 shown in FIG. 4. 
[FIG. 16] 

A flowchart explaining the operation of the learning apparatus shown in 
FIG. 15. 
[FIG. 17] 

A diagram illustrating an arrangement of primary-color filters. 
[FIG. 18] 

A diagram showing an arrangement of complement-color filters. 
[Explanation of Referenced Numerals] 
1 lens ; 3 CCD ; 6 image- signal processing circuit ; 
8 main CPU ; 31 defect-correcting circuit ; 35 interpolating section ; 
36 correction circuit ; 41 ADRC process circuit ; 
42 classification circuit ; 43 adaptation process circuit ; 
44 coefficient memory ; 61 learning apparatus ; 71 extraction circuit ; 

72 student-image block generating circuit ; 73 ADRC process circuit ; 
74 classification circuit ; 75 teacher-image block generating circuit ; 
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76 operating circuit ; 
79 coefficient memory 



77 learned data memory ; 78 operating circuit ; 
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[Document Name] ABSTRACT 
[Abstract] 

[Problems to be Solved] 

To generate a three-plate CCD output of high precision from an output of a 
single-plate CCD. 
[Means to Solve the Problems] 

Classification-adaptation process is performed on an output of a 
single-plate CCD, which represents pixels each having one of the three primary 
colors R, B and G, thereby to provide an output which is equivalent to one 
generated by a three-plate CCD and which represents pixels each having R, G and 
B components. Only the G component having the highest density is used as a 
class tap in the classification-adaptation process. 
[Selected Figure] FIG. 7 
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